Formulation of the learning problem, Part I

Now that we have seen an informal statement of the learning problem, as well as acquired some technical tools in the form of concentration inequalities, we can proceed to define the learning problem formally. Recall that the basic goal is to be able to predict some random variable Y of interest from a correlated random observation X , where the predictor is to be constructed on the basis of n i.i.d. training samples (X1,Y1), . . . , (Xn ,Yn) from the joint distribution of (X ,Y ). We will start by looking at an idealized scenario (often called the realizable case in the literature), in which Y is a deterministic function of X , and we happen to know the function class to which it belongs. This simple set-up will let us pose, in a clean form, the basic requirements a learning algorithm should satisfy. Once we are done with the realizable case, we can move on to the general setting, in which the relationship between X and Y is probabilistic and not known precisely. This is often referred to as the model-free or agnostic case. This order of presentation is, essentially, historical. The first statement of the learning problem is hard to trace precisely, but the “modern” algorithmic formalization seems to originate with the 1984 work of Valiant [Val84] on learning Boolean formulae. Valiant has focused on computationally efficient learning algorithms. The agnostic (or model-free) formulation was first proposed and studied by Haussler [Hau92] in 1992. In this lecture, I will be closely following the excellent exposition of Vidyasagar [Vid03, Ch. 3].